Online Learning in Medical Student Clerkship: A Survey of Student Perceptions and Future Directions

Background The coronavirus disease 2019 (COVID-19) pandemic had a major impact on medical education with clerkship students abruptly removed from clinical activities in 2020 and hastily immersed in online learning to maintain medical education. In 2022, students returned to in-person clinical experiences, but synchronous learning sessions continued online with extensive use of asynchronous online resources. This change offers a unique opportunity to gather information about students’ perspectives regarding the acceptability and effectiveness of online learning strategies. This study aims to explore the clerkship student experience with the integration of online learning and in-person learning into formalized educational sessions in clerkship. Methodology The authors administered an online survey to clerkship students at the Cumming School of Medicine at the University of Calgary, Canada in spring 2022. The survey consisted of primarily Likert-style questions to explore the perceived effectiveness of various online learning strategies. Results are reported as the proportion selecting “quite effective” or “extremely effective.” Results A total of 89 students responded to the survey (57.4% of graduating class). For synchronous online learning, case-based learning was perceived as the most effective teaching strategy (61.8%), and audience response systems were the most effective strategy for improving audience engagement (70.1%). For asynchronous online learning, interactive cases (84.9%) and student-developed online study guides (83.6%) were perceived as the most effective. Students held varying perceptions regarding how online learning impacted their well-being. When considering future clerkship curricula, the majority of clerkship students preferred a blend of in-person and online learning. Conclusions This study identified that most clerkship students prefer a hybrid of in-person and online learning and that ideal online learning curricula could include case-based learning, audience response systems, and a variety of asynchronous learning resources. These results can guide curriculum development and design at other medical institutions.


Introduction
The coronavirus disease 2019 (COVID-19) pandemic had dramatic effects on medical education.This impact was particularly evident in clerkship as students were abruptly removed from clinical activities during the initial months of the pandemic response (spring 2020).This rapid move to alternative educational methods was termed "Emergency Remote Teaching" (ERT).It employed extensive use of online learning to maintain medical education [1].Many studies have been published between 2020 and 2022 about the sudden transition to online learning for medical students with the purpose of maintaining access to education during a public health emergency [1][2][3].Starting in the fall of 2021, the world started transitioning out of the initial pandemic response, and students were gradually reintroduced to in-person learning.This shift allowed educators to re-evaluate the ongoing role of online learning in medical schools [4].
In contrast to ERT, there is a longer history of planned, purposeful online learning strategies in medical education [1].Synchronous online learning refers to live teaching sessions in which learners and teachers interact and exchange information simultaneously (e.g., video-conferenced lectures or small group sessions) [5].Asynchronous online learning refers to students using self-directed learning from online resources at their own pace (e.g., podcasts, videos, question banks) [5].The "flipped classroom" model is a hybrid of the two and involves asynchronous preparation followed by interactive synchronous discussion and teaching [6,7].Online learning has been shown to be similarly effective to traditional learning strategies, but significant controversy exists about when and how to best use it [8,9].
Senior medical students in clerkship participate in clinical rotations in a range of specialties.Traditionally, medical students in clerkship (sometimes referred to as "clerks" or "clinical clerks") have additional inperson structured teaching sessions tailored to each specialty rotation.At the time our study was conducted, clerkship-structured teaching at our institution remained mostly online despite clerks returning to fully inperson clinical rotations due to public health restrictions on large in-person group gatherings.
There may be a significant ongoing role for online learning in clerkship.At the time of conceptualizing this project, there was limited research about integrated online learning in clerkship after the initial stages of the pandemic.The experience and knowledge gained during the pandemic presents an opportunity to integrate lessons learned about online learning into future curricula.
Therefore, this study aimed to explore the clerkship student experience with online learning strategies integrated with in-person learning.The goal is to provide insights into the purposeful use of online learning in future clerkship curriculum development.This study focused on three main objectives: (1) to assess the perceived effectiveness of synchronous online learning strategies, (2) to assess the perceived effectiveness of asynchronous online learning strategies, and (3) to explore the impact of online learning on medical student well-being.

Study design and data collection methods
We developed a cross-sectional online survey informed by best practices in the medical education literature [10].Many survey questions were adapted from a recent program evaluation of ERT during the early stages of the COVID-19 pandemic [11].It consisted of the following four sections per our study objectives: Demographics, Synchronous Learning, Asynchronous Learning, and Wellness.The number of questions per section varied between 3 and 12 and consisted primarily of five-point Likert-style questions.Students rated the effectiveness of four different formats of synchronous online learning as well as three different learner engagement strategies (Table 1).Additionally, students also rated the effectiveness of multiple types of asynchronous online learning resources, including interactive online cases called "Cards," online readings, and student-made study guides.The survey was reviewed and revised by members of the research team which included clerkship students, residents, and staff physicians.The survey was also piloted with three recently graduated students (class of 2021) using concurrent probing and revised accordingly to improve content validity.The complete survey is available as Supplemental Digital Appendix 1.
The target population for this study was medical students in the Cumming School of Medicine, University of Calgary (in Alberta, Canada), class of 2022 (n = 156).This population was selected because these students uniquely experienced pre-pandemic in-person learning, pandemic online learning, and post-pandemic hybrid learning.To be eligible for participation, an individual needed to be a clerkship student at the Cumming School of Medicine with planned graduation in spring 2022.There were no exclusion criteria if participants met the previous criteria.
The survey was distributed online via Qualtrics®.Students were asked to rate only those learning strategies with which they had first-hand experience.

Data analysis
Data were analyzed with descriptive statistics using proportions.Gender and age differences in ratings of the effectiveness of synchronous and asynchronous learning strategies were compared using analysis of variance for gender and Pearson product-moment correlation for age.Gender and age comparisons of different synchronous and asynchronous teaching strategies could not be conducted due to wide variability in the number of students who had experience with each teaching strategy.As a total of 33 comparisons of the demographic variables and the ratings of effectiveness for various strategies were conducted, the Bonferroni correction was used to reduce the likelihood of committing a Type I error.The resulting level of significance was determined to be 0.05/33 = 0.0015.The results are, thus, considered significant if the probability values are below this value.Data analysis was conducted in SPSS version 29 (IBM Corp., Armonk, NY, USA).We used the consensus-based Checklist for Reporting of Survey Studies as a guideline for reporting the findings [12].

Synchronous online learning strategies
Participant ratings of synchronous online teaching strategies are shown in Table 2. Case-based learning was rated as the most effective for student learning, with 61.7% (50/81) of students rating it as quite or extremely effective.The second highest rating of quite or extremely effective was for online simulation (n = 34/72, 47.2%), followed by didactic lectures (n = 32/81, 39.5%).The lowest rating was for flipped classroom (n = 28/71, 39.4%).For student engagement strategies, audience response systems (e.g., polling) were viewed as the most effective, with 56/79 (70.9%) students rating them as quite or extremely effective.Breakout rooms (25/78, 32.1%) and collaborative documents (19/71, 26.8%) were viewed less favorably.There were no gender differences in these ratings (p > 0.0015).In terms of age, only one strategy was significant whereby younger students were likely to consider audience response systems to be effective for engagement (r = -0.41,p < 0.0015).When asked how many hours per week of synchronous teaching sessions participants would prefer in clerkship, the most frequent response was three to four hours per week (n = 43/81, 53.1%).Some students preferred five to eight hours per week (n = 14/81, 17.3%), and others preferred less at one to two hours per week (n = 21/81, 25.9%) or none at all (n = 3/81, 3.7%).When asked if it was easier to ask questions in online learning sessions than in person, there was no clear preference between either agreed or strongly agreed (29%) and disagreed or strongly disagreed (37%).

Asynchronous online learning resources
Table 3 summarizes the ratings of learning effectiveness for various asynchronous online learning resources.The most highly rated resources were Cards, with 62/73 (84.9%) students rating this strategy as quite or extremely effective, followed by student-made study guides (n = 51/61, 83.6%).The lowest rated resource was recorded lectures (n = 26/47, 55.3%).There were no gender or age differences in reported learning effectiveness (p > 0.0015).The majority of students appreciated protected time to study using online asynchronous learning resources, with 70/78 (89.7%) selecting agree or strongly agree.

Student well-being
Students reported a range of opinions when asked if online learning had increased their overall sense of wellness, with 29/78 (37.1%) selecting agree or strongly agree, and 23 (29.5%) selecting disagree or strongly disagree.Students reported that synchronous online learning sessions were more convenient to attend than in-person sessions, with 68/81 (84.0%) selecting agree/strongly agree.They also felt online learning was less disruptive to their clinical rotations, with 60/81 (74.1%) selecting agree or strongly agree.
Many students felt that the shift to online learning had increased their sense of social isolation, with 53/78 (67.9%) selecting agree or strongly agree.Most students selected disagree or strongly disagree when asked if online sessions provided a meaningful opportunity to connect with classmates (54/78, 69.2%) or instructors (51/78, 65.4%).

Overall teaching preferences
Participants were asked what model of learning they would prefer in clerkship if public health measures allowed.We found that 11/78 (14.1%) preferred all in-person learning, 22 (28.2%)preferred mostly in-person with some online learning, 28 (35.9%)preferred a similar amount of in-person and online learning, 14 (17.9%)preferred mostly online learning with some in-person learning, and 3 (3.8%)preferred all online learning.When asked if they were more likely to meet their learning objectives through online learning compared to in-person learning, 21/78 (26.9%) selected agree or strongly agree compared to 23 (29.5%) who selected disagree or strongly disagree.

Discussion
This cross-sectional, single-institution, survey-based study explored a unique group of students who experienced pre-pandemic, pandemic, and initial post-pandemic curriculum.To address this study's original objectives, (1) students perceived case-based learning as an effective learning strategy, especially when paired with audience response systems to enhance engagement; (2) students perceived Cards, or interactive online case-based questions, as an effective asynchronous learning strategy; and (3) students described that online learning, while convenient compared to in-person learning, also increases feelings of isolation.
Overall, it revealed that students prefer a mix of online and in-person structured teaching during clerkship.

Synchronous online learning
Our study highlights specific synchronous online learning strategies that students identified as more and less effective.Case-based learning was rated as the most effective learning synchronous strategy in this study.This result is similar to a recent systematic review, which showed that both students and instructors enjoy this type of instruction, perhaps because it promotes motivation for learning [14].Motivation for learning was particularly enhanced by the interactive component from both the learner and instructor involved in case-based learning.Furthermore, McLean's review of case-based learning in medical education indicates that when implemented with adequate participation from both educators and students, case-based learning can span all four stages of Kirkpatrick's model of learning that include enabling a positive reaction, acquiring knowledge, changing behaviors, and, finally, affecting practice behaviors [15,16].There are multiple published examples of case-based learning being successfully implemented online, including during the COVID-19 pandemic [14,17].
Audience response systems, such as Poll Everywhere (https://www.polleverywhere.com/)and Kahoot!quizzes (https://kahoot.com/academy/study/),were described in our study as an effective strategy for maintaining participation during synchronous sessions.Audience response systems have been shown to increase knowledge retention through retrieval practice and a previous review of audience response systems in post-secondary education shows that they improve student engagement, increase attendance, stimulate discussion, improve learning performance, and provide feedback to teachers [18][19][20].
Potential challenges for online synchronous learning include teleconferencing fatigue; research examining this shows that effective audience engagement may help circumvent fatigue [3].As such, audience response systems or other audience engagement strategies should be regularly implemented within synchronous online sessions.Additionally, our results suggest that students prefer limiting synchronous online learning to three to four hours per week, with the remaining time recommended for in-person activities or clinical experiences.

Asynchronous online learning
Students in this study identified a variety of effective online asynchronous learning strategies, with the most effective identified as Cards.
Cards is an open-access electronic resource developed by educators at the University of Calgary (https://cards.ucalgary.ca/).It uses automatic item generation to create a large number of multiple-choice questions to reflect different clinical presentations [21].Cards uses a variety of concepts to develop clinical decision-making skills, including identifying key features from patient cases to form a mental abstraction or problem representation, creating illness scripts, and using knowledge organization, storage, and retrieval theories [22].The Cards website currently has a library of over 350,000 cases [23].
Student-made study guides were also described as particularly effective.These guides include peer-reviewed student-made resources such as Toronto Notes (https://torontonotes.ca/)and Calgary Guide (https://calgaryguide.ucalgary.ca/)which provide short written and/or graphical summaries of major topics.
As discussed in a review examining study guides in medical education, peer-reviewed study guides provide structure to help with the identification of specific learning gaps and the development of further learning objectives [24].Study guides are a form of independent learning with the impression of receiving guidance from an experienced tutor [24].
We were surprised that some students found resources discovered online to be more valuable than resources provided by the program.We propose the following contributing factors: learners value autonomy in selfdirected learning, online resources are perceived as high quality, and word-of-mouth near-peer recommendation is powerful in spreading asynchronous online resources.Perhaps leaders in medical education could identify what resources their students are using, verify their quality and accuracy, and curate them to provide an array of resources tailored to course objectives.
Interestingly, students in this study did not rate flipped classroom sessions as particularly effective.There is widespread implementation of the flipped classroom type of pedagogy in medical education.The metaanalysis by Chen et al. on flipped classroom learning found that it is associated with high academic outcomes for theoretical knowledge compared to traditional didactic learning models [7].Conversely, there was no advantage for flipped classroom with regards to practical application such as in oral clinical examinations [7].Another systematic review of student perceptions of the flipped classroom model described that students enjoy the model and find it increases learning motivation and engagement [25].The discrepancies between our study and other studies may lie in the potential differences in how flipped classroom teaching is delivered at our institution or the lack of clarity around the definition of flipped classroom teaching for participants completing the survey.Additionally, it is possible that flipped classroom is less favorable in comparison to the variety of other learning strategies that students are provided.Many studies show that medical student well-being declined during the early stages of the COVID-19 pandemic; however, it is unclear whether online learning was a primary driver or contributor to this phenomenon [26][27][28][29].A benefit described by students relating to well-being was the convenience of online synchronous sessions.Clerkship involves participation in clinical learning, which makes this portion of training distinct from earlier years in medical training (pre-clerkship) where learning is primarily through traditional classroom methods.Clerkship rotations frequently use a distributed medical education model with students throughout different hospitals, clinics, and communities [30].Distributed models of education are well-suited to online learning strategies to provide equivalent experiences to students at different learning sites.Students described that having online synchronous sessions was less disruptive to their clinical time, thereby encouraging participation from any site and efficient return to clinical duties after sessions.Further literature supports that this convenience may allow for more meaningful clinical experience and professional identity formation, both of which have been demonstrated to be pillars of clerkship learning [31].

Wellness
Students described increasing social isolation with online learning.This phenomenon has been widely discussed [26,27].McInnerney described the inability to freely and immediately communicate with peers and instructors as the main factor in feeling isolated with online learning [32].This study describes that the utilization of synchronous communication before and during teaching sessions might be used in online learning to ameliorate feelings of isolation and form a sense of community [32].
The Association of Faculties of Medicine of Canada, which represents all faculties of medicine in Canada, has recently endorsed the importance of health-promoting learning environments and has adopted the Okanagan Charter (2015) [33].This charter was developed by a variety of stakeholders from more than 45 countries and the World Health Organization [34].It provides a framework for how we can continuously evaluate our work on well-being within medical education.Therefore, it is important that we assess the impact of online learning strategies on student well-being [35].
This study has several strengths.We surveyed a graduating class of students with a unique experience over three years, including starting pre-clerkship in-person, an abrupt transition to ERT, and a transition back to in-person clinical learning.These students experienced a range of online learning strategies and have valuable insights.We had a high response rate of 57% of the graduating class, which could be attributed to student interest in this topic, as well as an effective incentive structure.We also may be the first group to survey clerkship students about online learning after returning to full in-person clinical activities.
Regarding limitations, the nonprobability sample was obtained at a single site and may not reflect the experience of students at other institutions.Our survey was focused on student reactions to online learning strategies, which are the first level of the Kirkpatrick model of program evaluation [16].We acknowledge that the perceived effectiveness of learning from students might not reflect actual learning outcomes.For example, we did not investigate the impact on learning outcomes by assessing exam scores.Although this study focused on students' perceptions of effective online learning, it does not include other stakeholders' perspectives regarding online learning, such as educators.Finally, although we did develop the survey based on previous program evaluations and conducted a pilot test of our survey questions, we have only limited content validity of our scores.
Next steps include examining the effects of online learning on higher-level learning outcomes.Additionally, it is important to explore the effectiveness of current hybrid models of learning compared to previous inperson models.

2024
Patel et al.Cureus 16(2): e54541.DOI 10.7759/cureus.54541 This research was previously presented as a poster presentation at the Innovations in Medical Education Conference on February 16, 2023, and an oral presentation at the International Congress of Academic Medicine onApril 17, 2023.

Term Definition Synchronous Learning Live teaching
sessions where learners and teachers interact and exchange information simultaneously (e.g., video-conferenced lectures or small group sessions) Asynchronous Learning Using self-directed learning to gain new information from online resources at a learner's own pace (e.g., podcasts, videos, question

TABLE 3 : Perceived effectiveness of asynchronous online learning resources.
Given the wide availability of asynchronous resources online, the survey explored participants' use of resources provided by the clerkship and those discovered by students themselves.Most students used a mix of both, with 3/78 (3.8%) using only clerkship resources, 17 (21.8%)using mostly clerkship resources with few other resources, 31 (39.7%)reporting similar use of clerkship and other resources, 25 (32.1%) using mostly other resources with some clerkship resources, and 1 (1.3%) using only other resources.Many students reported that resources they discovered online were more useful than those provided by the clerkship rotation, with 38/78 (48.7%) selecting agree or strongly agree, 36 (46.2%) selecting neither agree or disagree, and 4 (5.1%)selecting disagree or strongly disagree.Frequently used asynchronous resources as indicated by students included UpToDate®, Toronto Notes (a peer-reviewed study guide developed by medical students at the University of Toronto, Canada that succinctly discusses major medical topics in point form), Osmosis, Anki, free open-access medical education podcasts, YouTube, and Calgary Guide (a peer-reviewed website that provides mind maps of a variety of medical topics developed by medical students and physicians).